Crowdsourcing the Creation of a Word–Emotion Association Lexicon

Even though considerable attention has been given to semantic orientation of words and the creation of large polarity lexicons, research in emotion analysis has had to rely on limited and small emotion lexicons. In this paper we show how the combined strength and wisdom of the crowds can be used to generate a large, high-quality, word–emotion association lexicon quickly and inexpensively. We flesh out various challenges in emotion annotation in a crowdsourcing scenario and propose solutions to address them. Most notably, in addition to questions about emotions associated with terms, we show how the inclusion of a word choice question can discourage malicious data entry, help identify instances where the annotator may not be familiar with the target term (allowing us to reject such annotations), and help obtain annotations at sense level (rather than at word level). We perform an extensive analysis of the annotations to better understand the distribution of emotions evoked by terms of different parts of speech. We identify which emotions tend to be evoked simultaneously by the same term and show that certain emotions indeed go hand in hand. We also analyze the polarity of terms (positive and negative), as well as what colors are associated with words. We find that associations with colors is directly correlated with the order that colors terms first came into existence in language. Also, red and black are strongly associated with negative emotion terms, whereas white and green are strongly associated with positive terms. The lexicon with close to 10,000 entries (one entry for each word–sense pair) will be made freely available.